Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add workflow to trigger OCP tests #12542

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

yuumasato
Copy link
Member

Description:

  • Leverage Content Test Filtering (CTF) to identify changes in OCP4 or RHCOS4 rules and automatically start Prow tests.
  • For every changed rule we elect a random OCP4 or RHCOS4 profile that selects it to be tested.
  • The list of elected profiles is de-duplicated and tests are run.
  • By default OCP 4.17 is used.
    • This could also be parametrized or randomized.

Rationale:

  • We often forget to test changes in OCP rules.

Review Hints:

  • Check that a comment was made by the workflow and Prow started testing profiles.
  • This is directly proposed to upstream repo to facilitate testing.

@yuumasato yuumasato added the OpenShift OpenShift product related. label Oct 25, 2024
Copy link

github-actions bot commented Oct 25, 2024

Start a new ephemeral environment with changes proposed in this pull request:

ocp4 (from CTF) Environment (using Fedora as testing environment)
Open in Gitpod

Fedora Testing Environment
Open in Gitpod

Oracle Linux 8 Environment
Open in Gitpod

@yuumasato yuumasato force-pushed the add_workflow_to_trigger_ocp_tests branch 6 times, most recently from 91760f6 to f677259 Compare October 25, 2024 19:15
Copy link

🤖 Trigger prow tests based on changed rules

/test 4.17-e2e-aws-ocp4-moderate-rev-4 /test 4.17-e2e-aws-ocp4-pci-dss-node

@yuumasato yuumasato force-pushed the add_workflow_to_trigger_ocp_tests branch 3 times, most recently from 5aa9f46 to 2a785c1 Compare October 25, 2024 21:02
Copy link

🤖 Trigger prow tests based on changed rules

/test 4.17-e2e-aws-ocp4-pci-dss-3-2
/test 4.17-e2e-aws-ocp4-pci-dss-node-3-2

Note: if a test is not started it can be that a CI Job is not configure for that particular profile or product.

Click here to see all the relevant profiles

Copy link

openshift-ci bot commented Oct 25, 2024

@github-actions[bot]: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test 4.12-e2e-aws-ocp4-cis
  • /test 4.12-e2e-aws-ocp4-cis-node
  • /test 4.12-e2e-aws-ocp4-e8
  • /test 4.12-e2e-aws-ocp4-high
  • /test 4.12-e2e-aws-ocp4-high-node
  • /test 4.12-e2e-aws-ocp4-moderate
  • /test 4.12-e2e-aws-ocp4-moderate-node
  • /test 4.12-e2e-aws-ocp4-pci-dss
  • /test 4.12-e2e-aws-ocp4-pci-dss-4-0
  • /test 4.12-e2e-aws-ocp4-pci-dss-node
  • /test 4.12-e2e-aws-ocp4-pci-dss-node-4-0
  • /test 4.12-e2e-aws-ocp4-stig
  • /test 4.12-e2e-aws-ocp4-stig-node
  • /test 4.12-e2e-aws-rhcos4-e8
  • /test 4.12-e2e-aws-rhcos4-high
  • /test 4.12-e2e-aws-rhcos4-moderate
  • /test 4.12-e2e-aws-rhcos4-stig
  • /test 4.12-images
  • /test 4.13-e2e-aws-ocp4-bsi
  • /test 4.13-e2e-aws-ocp4-bsi-node
  • /test 4.13-e2e-aws-ocp4-cis
  • /test 4.13-e2e-aws-ocp4-cis-node
  • /test 4.13-e2e-aws-ocp4-e8
  • /test 4.13-e2e-aws-ocp4-high
  • /test 4.13-e2e-aws-ocp4-high-node
  • /test 4.13-e2e-aws-ocp4-moderate
  • /test 4.13-e2e-aws-ocp4-moderate-node
  • /test 4.13-e2e-aws-ocp4-pci-dss
  • /test 4.13-e2e-aws-ocp4-pci-dss-4-0
  • /test 4.13-e2e-aws-ocp4-pci-dss-node
  • /test 4.13-e2e-aws-ocp4-pci-dss-node-4-0
  • /test 4.13-e2e-aws-ocp4-stig
  • /test 4.13-e2e-aws-ocp4-stig-node
  • /test 4.13-e2e-aws-rhcos4-bsi
  • /test 4.13-e2e-aws-rhcos4-e8
  • /test 4.13-e2e-aws-rhcos4-high
  • /test 4.13-e2e-aws-rhcos4-moderate
  • /test 4.13-e2e-aws-rhcos4-stig
  • /test 4.13-images
  • /test 4.14-e2e-aws-ocp4-bsi
  • /test 4.14-e2e-aws-ocp4-bsi-node
  • /test 4.14-e2e-aws-ocp4-pci-dss-4-0
  • /test 4.14-e2e-aws-ocp4-pci-dss-node-4-0
  • /test 4.14-e2e-aws-rhcos4-bsi
  • /test 4.14-images
  • /test 4.15-e2e-aws-ocp4-bsi
  • /test 4.15-e2e-aws-ocp4-bsi-node
  • /test 4.15-e2e-aws-ocp4-cis
  • /test 4.15-e2e-aws-ocp4-cis-node
  • /test 4.15-e2e-aws-ocp4-e8
  • /test 4.15-e2e-aws-ocp4-high
  • /test 4.15-e2e-aws-ocp4-high-node
  • /test 4.15-e2e-aws-ocp4-moderate
  • /test 4.15-e2e-aws-ocp4-moderate-node
  • /test 4.15-e2e-aws-ocp4-pci-dss
  • /test 4.15-e2e-aws-ocp4-pci-dss-4-0
  • /test 4.15-e2e-aws-ocp4-pci-dss-node
  • /test 4.15-e2e-aws-ocp4-pci-dss-node-4-0
  • /test 4.15-e2e-aws-ocp4-stig
  • /test 4.15-e2e-aws-ocp4-stig-node
  • /test 4.15-e2e-aws-rhcos4-bsi
  • /test 4.15-e2e-aws-rhcos4-e8
  • /test 4.15-e2e-aws-rhcos4-high
  • /test 4.15-e2e-aws-rhcos4-moderate
  • /test 4.15-e2e-aws-rhcos4-stig
  • /test 4.15-e2e-rosa-ocp4-cis-node
  • /test 4.15-e2e-rosa-ocp4-pci-dss-node
  • /test 4.15-images
  • /test 4.16-e2e-aws-ocp4-bsi
  • /test 4.16-e2e-aws-ocp4-bsi-node
  • /test 4.16-e2e-aws-ocp4-cis
  • /test 4.16-e2e-aws-ocp4-cis-node
  • /test 4.16-e2e-aws-ocp4-e8
  • /test 4.16-e2e-aws-ocp4-high
  • /test 4.16-e2e-aws-ocp4-high-node
  • /test 4.16-e2e-aws-ocp4-moderate
  • /test 4.16-e2e-aws-ocp4-moderate-node
  • /test 4.16-e2e-aws-ocp4-pci-dss
  • /test 4.16-e2e-aws-ocp4-pci-dss-4-0
  • /test 4.16-e2e-aws-ocp4-pci-dss-node
  • /test 4.16-e2e-aws-ocp4-pci-dss-node-4-0
  • /test 4.16-e2e-aws-ocp4-stig
  • /test 4.16-e2e-aws-ocp4-stig-node
  • /test 4.16-e2e-aws-rhcos4-bsi
  • /test 4.16-e2e-aws-rhcos4-e8
  • /test 4.16-e2e-aws-rhcos4-high
  • /test 4.16-e2e-aws-rhcos4-moderate
  • /test 4.16-e2e-aws-rhcos4-stig
  • /test 4.16-images
  • /test 4.17-e2e-aws-ocp4-bsi
  • /test 4.17-e2e-aws-ocp4-bsi-node
  • /test 4.17-e2e-aws-ocp4-cis
  • /test 4.17-e2e-aws-ocp4-cis-node
  • /test 4.17-e2e-aws-ocp4-e8
  • /test 4.17-e2e-aws-ocp4-high
  • /test 4.17-e2e-aws-ocp4-high-node
  • /test 4.17-e2e-aws-ocp4-moderate
  • /test 4.17-e2e-aws-ocp4-moderate-node
  • /test 4.17-e2e-aws-ocp4-pci-dss
  • /test 4.17-e2e-aws-ocp4-pci-dss-4-0
  • /test 4.17-e2e-aws-ocp4-pci-dss-node
  • /test 4.17-e2e-aws-ocp4-pci-dss-node-4-0
  • /test 4.17-e2e-aws-ocp4-stig
  • /test 4.17-e2e-aws-ocp4-stig-node
  • /test 4.17-e2e-aws-rhcos4-bsi
  • /test 4.17-e2e-aws-rhcos4-e8
  • /test 4.17-e2e-aws-rhcos4-high
  • /test 4.17-e2e-aws-rhcos4-moderate
  • /test 4.17-e2e-aws-rhcos4-stig
  • /test 4.17-images
  • /test e2e-aws-ocp4-bsi
  • /test e2e-aws-ocp4-bsi-node
  • /test e2e-aws-ocp4-cis
  • /test e2e-aws-ocp4-cis-node
  • /test e2e-aws-ocp4-e8
  • /test e2e-aws-ocp4-high
  • /test e2e-aws-ocp4-high-node
  • /test e2e-aws-ocp4-moderate
  • /test e2e-aws-ocp4-moderate-node
  • /test e2e-aws-ocp4-pci-dss
  • /test e2e-aws-ocp4-pci-dss-4-0
  • /test e2e-aws-ocp4-pci-dss-node
  • /test e2e-aws-ocp4-pci-dss-node-4-0
  • /test e2e-aws-ocp4-stig
  • /test e2e-aws-ocp4-stig-node
  • /test e2e-aws-rhcos4-bsi
  • /test e2e-aws-rhcos4-e8
  • /test e2e-aws-rhcos4-high
  • /test e2e-aws-rhcos4-moderate
  • /test e2e-aws-rhcos4-stig
  • /test images

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-ComplianceAsCode-content-master-4.12-images
  • pull-ci-ComplianceAsCode-content-master-4.13-images
  • pull-ci-ComplianceAsCode-content-master-4.14-images
  • pull-ci-ComplianceAsCode-content-master-4.15-images
  • pull-ci-ComplianceAsCode-content-master-4.16-images
  • pull-ci-ComplianceAsCode-content-master-4.17-images
  • pull-ci-ComplianceAsCode-content-master-images

In response to this:

🤖 Trigger prow tests based on changed rules

/test 4.17-e2e-aws-ocp4-pci-dss-3-2
/test 4.17-e2e-aws-ocp4-pci-dss-node-3-2

Note: if a test is not started it can be that a CI Job is not configure for that particular profile or product.

Click here to see all the relevant profiles

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@yuumasato yuumasato force-pushed the add_workflow_to_trigger_ocp_tests branch 3 times, most recently from 695fc80 to 4934c26 Compare October 25, 2024 22:39
Use CTF to identify changes in OCP rules, identify the profiles
selecting these rules, and start Prow tests.
@yuumasato yuumasato force-pushed the add_workflow_to_trigger_ocp_tests branch from 4934c26 to 84b6e16 Compare October 25, 2024 23:22
Not all profiles are configured in Prow.
This uses a hardcoded list of profiles to filter out the
"untestable" ones.
@yuumasato yuumasato force-pushed the add_workflow_to_trigger_ocp_tests branch from 84b6e16 to b7c47b0 Compare October 28, 2024 09:44
Copy link

🤖 Trigger prow tests based on changed rules

/test 4.17-e2e-aws-ocp4-high-node
/test 4.16-e2e-aws-ocp4-high-node
/test 4.17-e2e-aws-ocp4-moderate
/test 4.16-e2e-aws-ocp4-moderate

Note: if a test is not started it could be that a CI Job is not configure for that particular profile or product.

Click here to see all the relevant profiles
  • <OCP_VERSION>-e2e-aws-ocp4-bsi-node
  • <OCP_VERSION>-e2e-aws-ocp4-bsi
  • <OCP_VERSION>-e2e-aws-ocp4-cis-node
  • <OCP_VERSION>-e2e-aws-ocp4-cis
  • <OCP_VERSION>-e2e-aws-ocp4-e8
  • <OCP_VERSION>-e2e-aws-ocp4-high-node
  • <OCP_VERSION>-e2e-aws-ocp4-high
  • <OCP_VERSION>-e2e-aws-ocp4-moderate-node
  • <OCP_VERSION>-e2e-aws-ocp4-moderate
  • <OCP_VERSION>-e2e-aws-ocp4-pci-dss-4-0
  • <OCP_VERSION>-e2e-aws-ocp4-pci-dss-node-4-0
  • <OCP_VERSION>-e2e-aws-ocp4-pci-dss-node
  • <OCP_VERSION>-e2e-aws-ocp4-pci-dss
  • <OCP_VERSION>-e2e-aws-ocp4-stig

@yuumasato yuumasato requested a review from rhmdnd October 28, 2024 10:55
@yuumasato
Copy link
Member Author

@yuumasato yuumasato force-pushed the add_workflow_to_trigger_ocp_tests branch from b7c47b0 to 0cf1bbf Compare October 28, 2024 12:03
@yuumasato
Copy link
Member Author

I have dropped the artificial change: b7c47b0.
This is ready for review/merge.

if: ${{ steps.ctf.outputs.CTF_OUTPUT_SIZE != '0' && (contains(steps.product.outputs.prop, 'ocp4') || contains(steps.product.outputs.prop, 'rhcos4')) }}
id: profiles_to_test
run: |
OCP_VERSIONS=(4.17 4.16)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll need to manage this in addition to any OCP version changes in openshift/releases?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.
Another approach is to trigger the version-less tests, like e2e-aws-ocp4-high-node.

Sidenote: currently the version-less tests are running on 4.16, not the latest 4.17, 🙈

I considered running on two OCP versions, just to have more default coverage.
But it can be wasteful. I'm also okay with running only the version-less tests.

for rule in $RULES; do
readarray -t TEMP <<< $(grep -lr -e "- ${rule}\$" build/*/profiles | sort)

# Let's ilter out profiles for which we don't have a CI job configured
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: filter*


# Let's ilter out profiles for which we don't have a CI job configured
# Here is an example of how to quicly update this variable in the future
# TESTED_PROFILES=$(grep -r PROFILE= ./ComplianceAsCode-content-master__4.16.yaml | sort -u | sed 's/.*export PROFILE=\(.*\)/\1/')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This must be run from the openshift/releases repository, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a good work around for this.
We could clone the repo and parse the files, or issue a /test comment and parse the comment from the ci bot.

I added it because we don't have tests for all profile ids.
You can see in previous comments that pci-dss-3-2 was chosen for tests, but we don't have CI wired up for that.

The version-less jobs are expected to be running on latest OCP.
Let's directly parse the source of CI Job configuration to know what
profiles we can trigger jobs for.

Also, move it out of the rule iteration loop. We just need to define it
once.
These changes should be identified by CTF and Prow tests should be
started automatically.
Copy link

🤖 Trigger prow tests based on changed rules

/test e2e-aws-ocp4-high-node
/test e2e-aws-ocp4-moderate

Note: if a test is not started it could be that a CI Job is not configure for that particular profile or product.

Click here to see all the relevant profiles
  • <OCP_VERSION>-e2e-aws-ocp4-bsi-node
  • <OCP_VERSION>-e2e-aws-ocp4-bsi
  • <OCP_VERSION>-e2e-aws-ocp4-cis-node
  • <OCP_VERSION>-e2e-aws-ocp4-cis
  • <OCP_VERSION>-e2e-aws-ocp4-e8
  • <OCP_VERSION>-e2e-aws-ocp4-high-node
  • <OCP_VERSION>-e2e-aws-ocp4-high
  • <OCP_VERSION>-e2e-aws-ocp4-moderate-node
  • <OCP_VERSION>-e2e-aws-ocp4-moderate
  • <OCP_VERSION>-e2e-aws-ocp4-pci-dss-4-0
  • <OCP_VERSION>-e2e-aws-ocp4-pci-dss-node-4-0
  • <OCP_VERSION>-e2e-aws-ocp4-pci-dss-node
  • <OCP_VERSION>-e2e-aws-ocp4-pci-dss
  • <OCP_VERSION>-e2e-aws-ocp4-stig

@github-actions github-actions bot deleted a comment from openshift-ci bot Oct 29, 2024
Copy link

codeclimate bot commented Oct 29, 2024

Code Climate has analyzed commit 7b46c13 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 60.9% (0.0% change).

View more on Code Climate.

Copy link

openshift-ci bot commented Oct 29, 2024

@yuumasato: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/4.17-e2e-aws-ocp4-pci-dss-node f677259 link true /test 4.17-e2e-aws-ocp4-pci-dss-node
ci/prow/4.16-e2e-aws-ocp4-high-node b7c47b0 link true /test 4.16-e2e-aws-ocp4-high-node
ci/prow/4.17-e2e-aws-ocp4-high-node b7c47b0 link true /test 4.17-e2e-aws-ocp4-high-node
ci/prow/4.17-e2e-aws-ocp4-moderate b7c47b0 link true /test 4.17-e2e-aws-ocp4-moderate
ci/prow/e2e-aws-ocp4-high-node 7b46c13 link true /test e2e-aws-ocp4-high-node

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OpenShift OpenShift product related.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants